Continuous tracing and metric collection system

ABSTRACT

Techniques for capturing and recording processor events and scheduler data in a production system on a per processing resource basis are discussed herein. In some examples, a process metric collection component may be associated with the scheduler and the processing resource such that the process metric collection component can capture real time data associated with the processes or threads both executed by the processing resource and waiting to execute on the processing resource. The captured data may be used by the system to monitor operations.

BACKGROUND

Typically, software developers utilize a test environment and/or testhardware during development to identify and address any issues or bugsassociated with the product under development. Once testing is complete,the product is released or moved to a production environment. Theproduction environments are typically similar to the test environmentbut often does not include any testing or data recording components, asthe testing or data recording components consume processing resources,thereby reducing performance of the finished product. However, if anissue or bug is identified in the production environment, the developersare then tasked with reproducing the issue or bug in the testenvironment in order to identify and address the issue or bug. In somesituations, issue or bug correction may be time sensitive and/orrecreation of the issue or bug may be difficult, particularly in today'sincreasing complex production environments.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 is a process flow diagram illustrating an example data flow of aproduction system including integrated data and process metriccollection component, as described herein.

FIG. 2 is an example flow diagram illustrating an example processassociated with the production system of FIG. 1 , as described herein.

FIG. 3 is another flow diagram illustrating an example process of theproduction system of FIG. 1 , as described herein.

FIG. 4 is another flow diagram illustrating an example process ofassociated with monitoring the operations of a system with a processmetric collection component, as described herein.

FIG. 5 is an example pictorial diagram illustrating an examplearchitecture of the production system of FIG. 1 , as described herein.

FIG. 6 is a block diagram of an example system for implementing theproduction system of FIG. 1 , as described herein.

FIG. 7 is another block diagram of an example system for implementingthe production system of FIG. 1 , as described herein.

DETAILED DESCRIPTION

Techniques described herein are directed to a production system that mayinclude one or more data and process metric collection components. Insome examples, the data and process metric collection components may beconfigured to track and record process and scheduler related metrics foreach process that is executed by production system while the productionsystem is in operation. In some cases, the data and process metriccollection components are configured to track and record each process ofthe production system across multiple processing resources using aglobal clock. Thus, in the system discussed herein, if an issue occursduring operation, actual production data associated with each processingresource and each process is available to compare and analyze usingsynchronized time stamps without having to recreate the issue in a testenvironment.

In some examples, such as operations of an autonomous vehicle,recreation of a bug or issue in a test environment may be exceedinglydifficult as a reoccurrence of the bug or issue may depend on countlessever-adjusting and changing variables typical of real life physicalenvironment (e.g., system conditions, traffic conditions, weatherconditions, etc.). Additionally, in these types of examples, any delayin correcting the bug or issue may introduce uncertainty or may increaserisks for vehicle operation. Thus, the system discussed herein allowsfor identification and correction of bugs or issues using logs and datagenerated, analyzed, and/or stored during operation of the actualproduction environment. In some cases, the data and process metriccollection components, discussed herein, are able to track and recordthe production logs without causing a reduction in the performance ofthe overall system (e.g., the processing resources of the autonomousvehicle).

In some cases, each process metric collection component may beconfigured to send, via one or more networks, messages including thelogs or recorded data to a remote debugging system for storage. In someexamples, the process metric collection component may be integrated intoa message logging framework that translates the collected data into amessage that may be sent to the debugging system for aggregation, metriccalculation (e.g., cache misses, cycles per processor or instruction,etc.), and other processing. In this manner, the debugging system may beconfigured to aggregate the production data not only across multipleprocessing resources of the same device (e.g., vehicle) but also betweenprocessing resources of different devices (e.g., over the fleet ofvehicles). In some cases, the debugging system may generate variousgraphical user interfaces that may present the data and process metricsin various manners to programmers to assist with locating and correctingany issues identified. In some cases, the debugging system and/or thevehicle may be configured to identify the presence of a potentialissues. For example, an operating vehicle may initiate an emergency stopdue to a failure in a perception pipeline to timely perform objectrecognition, thereby preventing a potential collision. In this case, thevehicle or the debugging system may determine that the emergency stopwas due to a bug or issue in the code. In this case, the debuggingsystem may present, via a graphical user interface, the production dataassociated with a time span or window around the emergency stop.Allowing the programmer to identify any stalled, hung-up, or waitingprocesses in a time efficient manner.

In some implementations, the data and process metrics collected mayinclude the process itself, one or more time stamps corresponding to thetime at which the process was requested, a time stamp corresponding tothe time at which the process was initiated, a time stamp correspondingto the time at which the process was completed, priority of the process,and/or the particular processing resource that executed the process. Asa specific example, the data and process metrics collected may includeor be usable to determine instructions stalls, cycles per instruction,cache misses, bandwidth per socket or processing resource, threadcreation and destruction, priority of each process, among others.

In some examples, the data and process metric collection components maybe configured to log the process data based on scheduler events. Thus,unlike other conventional systems that may track data based on a fixedrate, the system discussed herein, may continuously log data for eachevent associated with the scheduler allowing for easier identificationof the particular process that is associated with the delay or issue. Inthis manner, each time the scheduler changes a process executing on aprocessing resource, the event may be logged with global time stampsthat may be used to correlate various process on various processingresources at a later date and to generate counters as part of thescheduler events.

In one specific example, the data and process metric collectioncomponents may be integrated into a kernel of a processing resource. Forexample, the data and process metric collection component may be loadedinto, for example, a Linux kernel using a Berkeley packet filter. Inthis manner, the data and process metric collection components may thenbe associated with the scheduler to capture the scheduler event data andcorrelate the scheduler events to the process execution on theprocessing resource.

In one specific example, the logs and recorded data may be used togenerate models or training data sets that may be used to simulate,test, and as input to train machine learned algorithms, as the logs andrecorded data represent real life operational situations. The models ortraining data sets can be used to improve a functioning of a computingdevice by providing additional data usable for performing subsequentoperations to control autonomous vehicles. For example, models ortraining data sets associated with image data can allow subsequentprocesses such as localization, perception (e.g., detecting,identifying, segmenting, classifying, tracking, etc.), route planning,trajectory generation, and the like to be performed more accurately, mayrequire less processing power, and/or may require less memory. Forexample, in some instances, faster and/or more accurate segmentation (orother processes) can be used in generating a trajectory of an autonomousvehicle, which can improve safety for occupants of an autonomousvehicle. These and other improvements to the functioning of the computerare discussed herein.

FIG. 1 is a process flow diagram illustrating an example data flow 100of a production system including integrated process metric collectioncomponents 102, as described herein. In the current example, a scheduler104 may be communicatively coupled to a processing resource 106 in orderto schedule processes on the processing resource 106 to execute as oneor more processes, illustrated as threads 108(1)-(N). Once each thread108(1)-(N) completes execution on the processing resource 106, theoutput of the thread 108 may be provided to or received by anothersystem or component, illustrated, herein, as system 110(1)-(M).

In the illustrated example, the process metric collection component 102may be communicatively coupled to or between the scheduler 104 and theprocessing resource 106 and between the processing resource 106 and theother systems 110. The process metric collection component 102 may alsobe configured to monitor, receive, or capture process or thread dataassociated with the threads 108 in progress. For instance, at variouscheckpoints the processing resource 106 may output a status or otherdata of each thread. In other instances, the process metric collectioncomponent 102 may output a status or other data at various triggersassociated with executing the threads 108, such as when a process startsa particular function or ends a particular function.

In one particular instance, the process metric collection component 102may be configured to incorporate into a kernel or operating systemassociated with the processing resources 106. In this example, theprocess metric collection component 102 may be stored in the same accessprotected memory space as the kernel or operating system code. In thismanner, the process metric collection component 102 is able to determineor identify each process or thread 108 scheduled or initiated by thescheduler 104 on the processing resource 106. Likewise, the processmetric collection component 102 is able to record the completion and/oroutput of each process or thread 108 by the processing resource 106.Thus, the process metric collection component 102 is able to collect,record, and/or log each process or thread 108 scheduled, initiated, andcompleted by the processing resource 106 during the operation of theautonomous vehicle.

In some examples, the process metric collection component 102 may alsohave access to schedule data of the scheduler 104, such that the processmetric collection component 102 is able to determine if any process iswaiting or being otherwise prevented from execution on the processingresource 106. In this manner, any delays associated with processes orthreads 108 failing to execute during any particular operationalconditions may be determined. As discussed above, the process metriccollection component 102 may be utilized with respect to processingresources 106 on an autonomous vehicle. In these examples, any delay inexecution even for milliseconds may result in a collision or injury to aperson. Thus, by implementing the process metric collection component102 with respect to the processing resources 106 of the autonomousvehicle, any issue or bugs may be traced using the operational data ofvehicle during operation reducing complexity or obliviating the need torecreate the bug in a lab on a test environment.

In the current implementation, the process metric collection component102 may be configured to receive a global clock signal from a globalclock 112, such that the process metric collection component 102 maygenerate time stamps associated with the execution of each thread 108based on the global time stamp rather than an internal clock of theprocessing resource. In this manner, the recorded or logged scheduledata and/or execution data generated by the process metric collectioncomponent 102 may be accurately analyzed and compared with log dataassociated with other processing resource of the autonomous vehicle. Forinstance, for safety purposes the autonomous vehicle may operate usingmultiple redundant computing device components. In this environment, ifa bug or other issue occurs, it may be important during debugging to beable to analyze the processes and threads 108 executing simultaneouslyand/or during a similar time window on the various compute components inorder to resolve the issue.

In the some examples, the process metric collection component 102 may beconfigured to cause a communication connection of the vehicle totransmit or send the logged or recorded data to a cloud-based debuggingsystem via one or more networks. For instance, the process metriccollection component 102 may record or log the schedule data andexecution data associated with the scheduler 104 and the processingresource 106 into a memory onboard the vehicle. The process metriccollection component 102 may also cause the recorded data (e.g., thestored schedule data and execution data) to be uploaded to thecloud-based debugging system on a periodic basis, in response to atrigger event (e.g., connection to the network or a predetermined amountof data being logged), or as a continuous stream.

In another specific example, the process metric collection component 102may be configured to generate processing thresholds or metrics based atleast in part on the logged or recorded data and to generate an alert ornotification, if a particular thread 108 exceeds the correspondingprocessing threshold or metric. For instance, if a particular thread 108typically completes execution within one to two milliseconds, theprocess metric collection component 102 may generate a correspondingprocess threshold to alert or notify an operator or particular system asto the delayed thread 108. In some specific cases, such as when thethread 108 is associated with an operation decision of an autonomousvehicle, the process metric collection component 102 may notify adecision making system of the vehicle that the thread has exceeded theexpected execution time and, thereby, cause the decision making systemto bring the vehicle to reduce speed or halt and/or notify repairpersonnel, operator, or passenger as to the delayed thread 108. In thismanner, the process metric collection component 102 may collect data andmonitor operations onboard and in substantially real time.

In some cases, the systems 110 may include various other processingsystems associated with, for example, an autonomous vehicle. Forinstance, the systems may include a perception system, a localizationsystem, a prediction system, a planning system, or other systems.

In the illustrated example, the processing resource 106 is shown as asingle processing resource. However, it should be understood that thescheduler 104 and the process metric collection component 102 may beconfigured to, respectively, schedule and log data associated withprocesses executed on one or more processing resources. In someimplementations, the processing resource 106 may be any suitableprocessor capable of executing instructions to process data and performoperations as described herein. By way of example and not limitation,the processing resource 106 may comprise one or more Central ProcessingUnits (CPUs), Graphics Processing Units (GPUs), or any other device orportion of a device that processes electronic data to transform thatelectronic data into other electronic data that can be stored inregisters and/or computer readable media. In some examples, integratedcircuits, gate arrays, field programmable gate arrays, and otherhardware devices can also be considered processors in so far as they areconfigured to implement encoded instructions.

FIG. 2 is an example flow diagram illustrating an example process 200associated with the production system of FIG. 1 , as described herein.As discussed above, data and process metric collection components may beconfigured to track and record process and scheduler related metrics foreach process or thread that is executed by production system while theproduction system is in operation. In some cases, the data and processmetric collection components are configured to track and record eachprocess of the production system across multiple processing resourcesusing a global clock, such that if an issue occurs during operation,actual production data associated with each processing resource and eachprocess is available to compare and analyze using synchronized timestamps without having to recreate the issue in a test environment.

At 202, a process metric collection component may receive schedule datafrom a scheduler. For instance, the schedule data may include variousprocesses, the processing resource assigned to execute the process, anyexecution or thread dependencies (e.g., thread completions required toinitiate the current process), etc.

At 204, the process metric collection component may receiveinitialization data associated with a thread or process initializing ona processing resource. For example, the process metric collectioncomponent may receive the initialization data from the scheduler ordetect the initialization on the processing resource.

At 206, the process metric collection component may receive a firstglobal time stamp associated with the initializing of the thread orprocess. For instance, as discussed above, the process metric collectioncomponent may be communicatively coupled to a global clock. The processmetric collection component may receive clock signals from the globalclock. In some cases, the global clock may be a high precision counterthat tracks time in specified units or cycles. The global clock may beconfigured to provide the same signals to each compute component orprocessing resource associated with the autonomous vehicle, such thateach compute component and each process metric collection component maylog process or thread data using the global time stamp.

In another example, the process metric collection component may receivea local time stamp from a local clock associated with the processingresource executing the thread or process. The process metric collectioncomponent may initially include the local time stamp as the time theprocessing resource begins execution of the process or thread. In thisexample, at various periodical intervals, the process metric collectioncomponent may receive substantially simultaneously a reference globaltime stamp from the global clock and a reference local time stamp fromthe local clock. The process metric collection component may thendetermine a difference between the reference global time stamp and thereference local time stamp. Then the process metric collection componentmay adjust the local time stamp assigned to the process or thread by thedifference. In this manner, the process metric collection component mayutilize the local clock when recording the data associated with aprocess and then adjust the time stamps of the collected data at variousintervals such that a global time stamp is still available when thecaptured process metrics and data is reviewed.

At 208, the process metric collection component may receive executiondata associated with the thread or process as the thread or processexecutes on the processing resource. For instance, at variouscheckpoints the processing resource may output a status or other data ofeach thread being executed. In other instances, the process metriccollection component 102 may output a status or other data at varioustriggers associated with executing the threads 108, such as when aprocess starts a particular function or ends a particular function.

At 210, the process metric collection component may receive a secondglobal time stamp associated with the execution of the thread orprocess. In this manner, the process metric collection component may beable to associate the second time stamp the checkpoint or trigger andthe execution data output by the processing resource. As discussedabove, in an alternative example, the process metric collectioncomponent may again initially utilize a local time stamp from a localclock associated with the processing resource executing the thread orprocess to generate the time stamp for the captured data and then updateor adjust the local time stamp using a comparison of an output of alocal clock and an output of the global clock.

In the current example, the process 200 shows one instance of theprocess metric collection component capturing execution data while athread or process is executed by the processing resource. However, itshould be understood, that the process metric collection component maycapture execution data at multiple times during the execution of asingle thread and that during each instance a time stamp may be appliedto the execution data.

At 212, the process metric collection component may determine acompletion of the process by the particular processing resource. Forinstance, the process metric collection component may receive completiondata associated with the thread or process executed by the processingresource. For example, the process metric collection component mayreceive the completion data from the scheduler or detect the completionon the processing resource.

At 214, the process metric collection component may receive a thirdglobal time stamp associated with the completion of the thread orprocess. In this manner, the process metric collection component may beable to associate the first time stamp with the initialization of thethread or process, the second time stamp associated with the execution,and the third time stamp with the completion of the thread or processand, thus, log the start time, end time, and an execution time spanassociated with the thread or process.

As discussed above, in an alternative example, the process metriccollection component may initially utilize a local time stamp from alocal clock associated with the processing resource executing the threador process to generate the time stamp for the captured data and thenupdate or adjust the local time stamp using a comparison of an output ofa local clock and an output of the global clock.

At 216, the process metric collection component may generate a messageassociated with the thread or process. For example, the process metriccollection component may aggregate data associated with each thread orprocess completed during a predetermined time period into a message ormessage that may be sent to a remote debugging system. In some cases,the process metric collection component may format the scheduler databased on a designated format for the message protocol. In some cases,the process metric collection component may be configured to generatethe message or message based on certain criteria being meet or exceeded.In these cases, if the criteria have not been met or exceeded, theprocess 200 may return to 204 and, the process metric collectioncomponent may record additional data associated with an additionalthread or process.

At 218, the process metric collection component may cause the message tobe sent to a remote system. For example, the process metric collectioncomponent may cause the message or message to be sent to a cloud-baseddebugging system. In some cases, the uploading of the message may beperformed on a periodic basis, in response to a trigger event, or as acontinuous stream.

In the current example, the process 200 sends a single message at 218,however, it should be understood that in some implementations, messagesmay be sent to the remote system at periodic intervals, in response tovarious triggers (e.g., and amount of data collected), or, even in somecases, upon completion of capture of each instance of data (e.g., aftersteps 206, 210, and 214 in the current example).

In the current example, the process 200 is discussed with respect to asingle thread or process executing on the processing resource. However,it should be apparent to one skilled in the art that the processingresource may have multiple threads or processes actively executing atthe same time and that the process metric collection component mayrecord data associated with each thread or process substantiallysimultaneously.

FIG. 3 is another flow diagram illustrating an example process 300 ofthe production system of FIG. 1 , as described herein. As discussedabove, process metric collection components may be integrated into thekernel or operating system of processing resources onboard productionautonomous vehicles 302 to record and log scheduler data and process orthread execution data as the processing resources execute instructionsin actual operating environments.

In the illustrated example, the process metric collection components ofthe vehicles 302 may be configured to send or otherwise transmitmessages 304 to a remote debugging system 306 via one or more wirelessnetworks. The messages 304 may include log files, scheduler data, and/orprocess execution data for each processing resource on each vehicle 302.For example, in some cases, the vehicles 302 may include multipleredundant compute units for use to compare results and, thereby,determine if an issue is occurring and/or ensure the safety of anyindividuals riding in the vehicles 302. In other cases, the vehicles 302may include drive units each having its own compute units, such thateither end of the vehicle 302 may act as the front of the vehicle 302based on a direction of travel.

At any time during operation, any of the vehicles 302 may generate anissue report 308. The issue report 308 may be generated by the processmetric collection component or any other component associated with theoperations of the vehicle 302. In other examples, issues that cause thegeneration of an issue report 308 may include but are not limited todetection of software performance or latency problems, failure of one ormore process to execute within an expected threshold of time (or aprocess missing a deadline). In other examples, an issue may include aplanning component or other decision making component may encounter asituation in which the vehicle 302 is forced to stop for the safety ofthe individuals riding in the vehicle 302, such as when the planningcomponent is unable to make a decision, the planning component isreceiving inconsistent data from one or more other systems, a locationof an object present in the environment is unknown, the planningcomponent is unable to predict a motion path of an object, among variousother situations. In these cases, the vehicle 302 may come to a stop andsend an issue report 308 that is received by the debugging system 306.In some cases, the issues may be user defined. Various types ofscheduler and event data that may be recorded by the process metriccollection component and sent as part of a message 304 as well assituations that may trigger an issue report can be found in patentapplication Ser. No. 16/224,385 entitled “Event-Based Data Logging” andfiled on Dec. 18, 2018, which is incorporated herein by reference in itsentirety.

Once an issue report 308 is received from a vehicle 302, the debuggingsystem 306 may perform issue identification 310. For instance, in oneexample, the debugging system 306 may parse the issue report 308 toidentify a time stamp associated with the cause of the issue report 308.As discussed above, the process metric collection component of thevehicle 302 is configured to time stamp the data within the message 304using a global clock signal. Thus, the issue report 308 may also includea time stamp generated from the global clock signal even when adifferent component or system generated the issue report 308. Thus, thedebugging system 306 may extract the vehicle 302 a vehicle identifierand a time stamp from the issue report 308. The debugging system 306 maythen perform a log file section 312 based on the vehicle identifier. Insome situations, such as when a processing resource associated with theissue report 308 is known, the debugging system 306 may also select logfiles based on the particular processing resource to further reduce theoverall amount of data presented to the programmer.

The debugging system may also identify a time window identification 314within the message 304 that includes the time indicated by the timestamp in the issue report 308 and reduce the overall amount of data byperforming log file reduction 316 using the time window 314. Asdiscussed above, the message 304 may include all of the scheduler dataand process execution data of multiple processing resources onboard thevehicle 302 resulting in the amount of overall data within each of themessage 304 being very large. For instance, in some cases, each message304 may contain in the range of approximately 300 megabytes to 400megabytes per minute. Thus, the debugging system 306 may select the logfiles by applying a time window as small as five milliseconds centeredaround the time stamp of the issue report 308.

Once the log files are limited to a number that may be parsed orexamined by a programmer, the debugging system 306 may present the logfiles on a debugging interface 318. The programmer may review the logs,detect bugs, update the software code if necessary, and perform testingon the new software. For example, the log files and/or execution datamay also be used to generate and/or train a model that may be used totest the new or edited code prior to shipping to a production system.Once the software update 320 is debugged and tested, the software update320 may be pushed or sent to the vehicles 302 to prevent the issue fromreoccurring during operation.

FIG. 4 is another flow diagram illustrating an example process 400 ofassociated with monitoring the operations of a system with a processmetric collection component, as described herein. In some cases, thesystem, such as an autonomous vehicle, may utilize historical processmetrics and data collected over a period of time to generate per processtime thresholds associated with the execution of each or selectprocesses being executed by a processing resources. In some examples,the thresholds may be utilized to determine if a process is stalled,waiting, or experience other issues (e.g., a hardware problem).

At 402, the process metric collection component and/or a monitoringcomponent may generate a per process threshold based at least in part onhistorical process metrics and data. In some cases, the historicalprocess metrics and data may be data collected by the process metriccollection component for the corresponding process or thread on thespecific processing resource, such that the threshold range may betailored to past performance on an individualized process and resourcebasis. In other cases, the historical process metrics and data may begeneric to the type of system, process, or resource and collected andcomplied across multiple systems. In some cases, the per processthreshold may be a range of thresholds. For instance, if the system is avehicle, then the range of thresholds may be based on a speed orvelocity of the vehicle, a number of objects within a predefineddistance of the vehicle, presence of a passenger, or a combinationthereof. As one particular example, the threshold time may increase asthe speed of the vehicle is reduced.

At 404, the process metric collection component and/or a monitoringcomponent may receive execution data from a process being executed on aprocessing resource and, at 406, the process metric collection componentor a monitoring component may determine a current processing timeassociated with the process based at least in part on the execution dataand at least one time stamp received from a global and/or local clock.For instance, the execution data may include a status, an indication ofthat the process has achieved a particular check point, or a change infunction. As discussed above, the process metric collection componentmay also receive a time stamp from a global clock or local clock thatcan be used to determine the current processing time by comparing to thetime stamp associated with the initialization of the process.

At 408, the process metric collection component and/or a monitoringcomponent may determine if the execution time is greater than acorresponding per process threshold. If the execution time is less thanthe per process threshold, the process 400 may return to 404 as theprocess is within normal and expected operating parameters. However, ifthe execution time is greater than the per process threshold (or the perprocess threshold range—the threshold value at the current speed orvelocity), than the process 400 may advance to 410.

At 410, the process metric collection component and/or a monitoringcomponent may send an alert. For example, the alert may cause anothersystem of the vehicle to change an operational state or status of thevehicle, reduce the velocity of the vehicle, and/or bring the vehicle toa stop. In other cases, the alert may be sent to an operator (eitherlocal or remote) to allow the remote operator to take control of theoperations of the vehicle or to a maintenance personnel such that thevehicle can be marked for repair or investigation upon return.

In some cases, the process 400 may also be used to determine if thesystem (e.g., vehicle) is operating within normal operations or at anominal level. For instance, the per process thresholds may beconfigured to represent a minimum level or nominal level of operationsexpected out of the system. In some cases, the normal operation mayinclude determining that a process starting and stopping without anyexceptions, no faulty memory calls, processes not being terminated earlyor late (e.g., finishing on time), processing resource and/or memoryutilization is a threshold percentage, no disengagements of a systemoccurred during processing, etc.

FIG. 5 is an example pictorial diagram illustrating an examplearchitecture 500 of the production system of FIG. 1 , as describedherein. In the illustrated example, a Kernel 502 associated with aprocessing resource 504 is shown. In this example, the process metriccollection component 506 is configured to receive data from thescheduler 408 of the Kernel 502 as well as to record or captureinitializations and completions of threads and or processes on theprocessing resource 504. For instance, in the illustrated example, theprocess metric collection component 506 may be configured to record datasent between the scheduler 508 and the processing resource 504 andbetween the processing resource 504 and a short-term memory 510 (e.g.,one or more caches).

FIGS. 6 and 7 illustrate example systems for implementing the techniquesdescribed herein, in accordance with embodiments of the disclosure. Insome examples, the systems may include one or multiple features,processing resources, components, and/or functionality of embodimentsdescribed herein with reference to FIGS. 1-4 . As discussed above, insome embodiments, the systems may include autonomous vehicles.

FIG. 6 is a block diagram of an example system 600 for implementing theproduction system of FIG. 1 , as described herein. In this embodiment,the system 600 is an autonomous vehicle 602 that may include a vehiclecomputing device 604, multiple processing resources 606(1)-(N), one ormore communication connections 608, a global clock 610, computerreadable media 612, as well as other devices 614 (e.g., sensor systems,navigation systems, drive systems, or other systems associated with anautonomous vehicle).

As discussed above, the vehicle computing device 604 may include one ormore processing resources 606. Each of the processing resources606(1)-(N) may be associated with a scheduler 616(1)-(M) and a processmetric collection component 618(1)-(K). The process metric collectioncomponents 618(1)-(K) may be configured to track and record scheduler616 related metrics for each process or thread that is executed bycorresponding processing resource 606 while the vehicle 602 is inoperation.

In some cases, the process metric collection components 618(1)-(K) maybe configured to receive a global clock signal from the global clock 610such that the recorded data may be time stamped and later comparedacross multiple processing resources 606. Thus, if an issue occursduring operation of the vehicle 602, actual production data associatedwith each processing resource 606 and each process or thread isavailable to compare and analyze using synchronized time stamps withouthaving to recreate the issue in a test environment.

In some case, the process metric collection components 618(1)-(K) may beconfigured to send over one or more networks the logs or messages to aremote debugging system for storage via the communication connections608. For example, the communication connection(s) 608 may facilitatecommunication with other remote computing device(s) such as thedebugging system. For instance, the communications connection(s) 608 mayenable Wi-Fi-based communication such as via frequencies defined by theIEEE 802.11 standards, short range wireless frequencies such asBluetooth®, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.)or any suitable wired or wireless communications protocol that enablesthe respective computing device to interface with the other computingdevice(s).

Additionally, in some example, the process metric collection components618(1)-(K) may be configured to generate processing thresholds ormetrics based at least in part on the captured process metrics and data626 and to generate an alert or notification, if a particular threadexceeds the corresponding processing threshold or metric. For instance,if a particular thread typically completes execution within one to twomilliseconds, the process metric collection components 618(1)-(K) maygenerate a corresponding process threshold to alert or notify anoperator or particular system as to the delayed thread. In someexamples, the thresholds used to determine if an alert or notificationshould be sent may be based at least in part on a speed or velocity ofthe vehicle 602. For instance, as the vehicle 602 increases in speed orvelocity the processing thresholds may be reduced as a delay of 0.5milliseconds at 15 miles per hour may be acceptable but a delay of 0.1milliseconds at 60 miles per hour may not.

In the current example, each of the processing resources 606(1)-(N) maybe associated with a corresponding scheduler 616(1)-(M) and acorresponding process metric collection component 618(1)-(K). However,in other examples, a global scheduler 616 and a global process metriccollection component 618 may be associated with the processing resources606.

The computer readable media 612 may be communicatively coupled with theone or more processing resources 606(1)-(N). In the illustrated example,the computer readable media 612 of the vehicle computing device 604 maystore monitoring systems 620 as well as other systems 622 (such as aperception system, a localization system, a prediction system, aplanning system, navigation system, system controllers, etc.). Thecomputer readable media 612 may also store data, such as sensor data624, process metrics and data 626 recorded by the process metriccollection components 618(1)-(K) (e.g., log files, event data, schedulerdata, execution data, etc.), and threshold data 628, such as a thresholdassociated with an expected execution time of various processes andthreads.

In some examples, in addition to or in lieu of the process metriccollection components 618 monitoring the execution of the processes andthreads, the monitoring system 620 may also be configured to generateprocessing thresholds or metrics based at least in part on the capturedprocess metrics and data 626 and the speed or velocity of the vehicle602. In these examples, the monitoring system 620 may generate the alertor notification, if a particular thread or process exceeds thecorresponding processing threshold or metric for the particular threador process (or system or subsystem) at the vehicle's 602 current speed.In this manner, the process metric collection components 618 and themonitoring system 620 may together monitor operations onboard thevehicle 602 and detect any potential problems in substantially realtime.

FIG. 7 is another block diagram of an example system 700 forimplementing the production system of FIG. 1 , as described herein. Insome embodiments, the system 700 may include a vehicle 702. The vehicle702 may include a vehicle computing device 704, one or more sensorsystems 706, one or more communication connections 708, and one or moredrive systems 710.

The vehicle computing device 704 may include one or more processors 712(or processing resources) and computer readable media 714communicatively coupled with the one or more processors 712. In theillustrated example, the vehicle 702 is an autonomous vehicle; however,the vehicle 702 could be any other type of vehicle, or any other system(e.g., a robotic system, a camera enabled smartphone, etc.). In theillustrated example, the computer readable media 714 of the vehiclecomputing device 704 stores a scheduling system 716, a process metriccollection component 618, global clock system 620, monitoring system 722as well as other systems (e.g., planning system, object predictionsystem, or other system associated with an autonomous vehicle). Thecomputer readable media 714 may also store sensor data 724, processmetrics and data 726 (e.g., log files, scheduler data, event data,executing data, etc.), and threshold data (e.g., process thresholdsbased on historical process metrics and data 726 and operationalparameters, such as a speed or velocity, of the vehicle 702). Thoughdepicted in FIG. 6 as residing in computer readable media 714 forillustrative purposes, it is contemplated that the scheduling system 716and the process metric collection system 718 may be integrated with theprocessor 712, as illustrated above with respect to FIG. 6 . In someimplementations, it should be understood that the systems as well asdata stored on the computer readable media may additionally, oralternatively, be accessible to the vehicle 702 (e.g., stored on, orotherwise accessible by, other computer readable media remote from thevehicle 702).

In at least one example, the scheduling system 716 may be configured toschedule and/or assign processes or threads to be performed on theprocessors 712. For example, the scheduling system 716 may manage threador process dependencies, load balancing, resource sharing, throughput,thread or process wait time, etc. In some cases, the scheduling system716 may include multiple schedulers, such as a process scheduler,long-term scheduler, admission scheduler, medium-term scheduler, CPUscheduler, etc.

In some cases, the process metric collection system 718 may becommunicatively coupled to or between the scheduling system 716 and theprocessors 712 and between the processors 712 and the other systems 720.In one particular instance, the processing metric collection component718 may be incorporated into a kernel or operating system associatedwith the processors 712. In this example, the process metric collectionsystem 718 may be stored in the same access protected memory space ofthe computer readable media 714 as the Kernel or operating system code.In this manner, the process metric collection system 718 is able todetermine or identify each process or thread scheduled or initiated bythe scheduling system 716 to execute on the processors 712. Likewise,the process metric collection system 718 is able to record thecompletion and/or output of each process or thread by the processors712. Thus, the process metric collection system 718 is able to collect,record, and/or log each process or thread scheduled, initiated, andcompleted by the processors 712 during the operation of the autonomousvehicle 702.

In some examples, the process metric collection system 718 may also haveaccess to schedule or event data, such that the process metriccollection system 718 is able to determine if any process is waiting orbeing otherwise prevented from execution on the processors 712 by thescheduling system 716. In this manner, any delays associated withprocesses or threads failing to execute during any particularoperational conditions may be determined.

As discussed above, the process metric collection system 718 may beutilized with respect to processors 712 on the vehicle 702. In theseexamples, any delay in execution even for milliseconds may result inincreased risk to vehicle(s) or objects in an environment person. Thus,by implementing the process metric collection component 618 with respectto the processors 712 of the production vehicle 702, any issue or bugsmay be traced using the operational data of vehicle 702 duringoperation, thereby reducing any need to recreate the bug on a testenvironment.

In the current example, the global clock system 710 may be configured toprovide a uniform clock signal to the various systems of the vehicle 702including the process metric collection system 718. In this manner, theprocess metric collection system 718 may generate time stamps associatedwith the execution of each thread or process based on the global timestamp rather than an internal clock of a particular processor 712. Inthis manner, the recorded or logged schedule data and/or execution datagenerated by the process metric collection system 718 may be accuratelyanalyzed and compared with log data associated with other processors 712of the vehicle 702.

In some examples, the monitoring system 722 may also be configured togenerate processing thresholds data 742 based at least in part on thecaptured process metrics and data 726 and operational parameters of thevehicle 702, such as a speed or velocity. In these examples, themonitoring system 722 may generate the alert or notification to varioussystems of the vehicle 702, such as the planning systems, perceptionsystems, prediction systems, or drive systems 710, if a particularthread or process exceeds the corresponding processing threshold data726 at the vehicle's 702 current operational parameters. In this manner,the process metric collection components 718 and the monitoring system722 may together monitor operations onboard the vehicle 702 and detectany potential problems in substantially real time.

The vehicle 702 can also include one or more communication connection(s)708 that enable communication between the vehicle 702 and one or moreother local or remote computing device(s). For instance, thecommunication connection(s) 708 may facilitate communication with otherlocal computing device(s) on the vehicle 702 and/or the drive system(s)710. Also, the communication connection(s) 708 may allow the vehicle 702to communicate with other nearby computing device(s) (e.g., other nearbyvehicles, traffic signals, etc.). The communications connection(s) 708also enable the vehicle 702 to communicate with remote teleoperationscomputing device or other remote services.

The communications connection(s) 708 may include physical and/or logicalinterfaces for connecting the vehicle computing device 704 to anothercomputing device (e.g., computing device(s) 730) and/or a network, suchas network(s) 728. For example, the communications connection(s) 708 mayenable Wi-Fi-based communication such as via frequencies defined by theIEEE 802.11 standards, short range wireless frequencies such asBluetooth®, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.)or any suitable wired or wireless communications protocol that enablesthe respective computing device to interface with the other computingdevice(s).

In some examples, the process metric collection system 718 may beconfigured to cause the communication connections 708 of the vehicle 702to transmit or send the logged or recorded data (e.g., the processmetrics and data 726) to the computing device(s) 730. In some instances,the process metric collection system 718 may then cause the processmetrics and data 726 to upload to the computing devices 730 on aperiodic basis, in response to a trigger event (e.g., connection to thenetwork 728 or a predetermined amount of data being logged), or as acontinuous stream.

In at least one example, the sensor system(s) 706 can include lidarsensors, radar sensors, ultrasonic transducers, sonar sensors, locationsensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertialmeasurement units (IMUs), accelerometers, magnetometers, gyroscopes,etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.),microphones, wheel encoders, environment sensors (e.g., temperaturesensors, humidity sensors, light sensors, pressure sensors, etc.), andone or more time of flight (ToF) sensors, etc. The sensor system(s) 706can include multiple instances of each of these or other types ofsensors. For instance, the lidar sensors may include individual lidarsensors located at the corners, front, back, sides, and/or top of thevehicle 702. As another example, the camera sensors can include multiplecameras disposed at various locations about the exterior and/or interiorof the vehicle 702. The sensor system(s) 706 may provide input to thevehicle computing device 704. Additionally, or alternatively, the sensorsystem(s) 706 can send sensor data, via the one or more networks 728, tothe one or more computing device(s) at a particular frequency, after alapse of a predetermined period of time, in near real-time, etc.

In at least one example, the vehicle 702 can include one or more drivesystems 710. In some examples, the vehicle 702 may have a single drivesystem 710. In at least one example, if the vehicle 702 has multipledrive systems 710, individual drive systems 710 can be positioned onopposite ends of the vehicle 702 (e.g., the front and the rear, etc.).In at least one example, the drive system(s) 710 can include one or moresensor systems 706 to detect conditions of the drive system(s) 710and/or the surroundings of the vehicle 702, as discussed above. By wayof example and not limitation, the sensor system(s) 706 can include oneor more wheel encoders (e.g., rotary encoders) to sense rotation of thewheels of the drive systems, inertial sensors (e.g., inertialmeasurement units, accelerometers, gyroscopes, magnetometers, etc.) tomeasure orientation and acceleration of the drive system, cameras orother image sensors, ultrasonic sensors to acoustically detect objectsin the surroundings of the drive system, lidar sensors, radar sensors,etc. Some sensors, such as the wheel encoders may be unique to the drivesystem(s) 710. In some cases, the sensor system(s) 706 on the drivesystem(s) 710 can overlap or supplement corresponding systems of thevehicle 702.

In at least one example, the components discussed herein can processsensor data 724, as described above, and may send their respectiveoutputs, over the one or more network(s) 728, to one or more computingdevice(s) 730. In at least one example, the components discussed hereinmay send their respective outputs to the one or more computing device(s)730 at a particular frequency, after a lapse of a predetermined periodof time, in near real-time, etc.

In some examples, the vehicle 702 can send sensor data to one or morecomputing device(s) 730 via the network(s) 728. In some examples, thevehicle 702 can send raw sensor data 724 to the computing device(s) 730.In other examples, the vehicle 702 can send processed sensor data 724and/or representations of sensor data (for instance, the objectperception tracks) to the computing device(s) 730. In some examples, thevehicle 702 can send sensor data 724 to the computing device(s) 730 at aparticular frequency, after a lapse of a predetermined period of time,in near real-time, etc. In some cases, the vehicle 702 can send sensordata (raw or processed) to the computing device(s) 730 as one or morelog files.

The computing device(s) 730 may include processor(s) 732 and computerreadable media 734 storing a debugging component 736, a process metricanalysis component 738, as well as process metrics and data 726 receivedfrom the vehicle 702 and debug data 740. In some examples, the debuggingcomponent 736 may include a debugging interface that may be used by oneor more programmers to trace any issues via the process metrics and data726 and to thereby edit or update the software of one or more systems722 operating on the vehicle 702. In some cases, the process metrics anddata 726 presented to the programmers via the interface may be limitedto a time window and/or to processes associated with one or moreparticular processor 712.

In some implementations, the process metric analysis component 738 maybe configured to generate various metrics (e.g., cache misses, averagewait, average throughput, available bandwidth, etc.). In some cases, theprocess metric analysis component 738 may be configured to generate oneor more models from the process metrics and data 726 that may be usedfor machine learning and/or future code testing.

The processor(s) 712 of the vehicle 702 and the processor(s) 732 of thecomputing device(s) 730 may be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)712 and 732 can comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that can be stored in registers and/orcomputer readable media. In some examples, integrated circuits (e.g.,ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardwaredevices can also be considered processors in so far as they areconfigured to implement encoded instructions.

Computer readable media 714 and 734 are examples of non-transitorycomputer-readable media. The computer readable media 714 and 734 canstore an operating system and one or more software applications,instructions, programs, and/or data to implement the methods describedherein and the functions attributed to the various systems. In variousimplementations, the computer readable media can be implemented usingany suitable computer readable media technology, such as staticrandom-access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of computer readablemedia capable of storing information. The architectures, systems, andindividual elements described herein can include many other logical,programmatic, and physical components, of which those shown in theaccompanying figures are merely examples that are related to thediscussion herein.

As can be understood, the components discussed herein are described asdivided for illustrative purposes. However, the operations performed bythe various components can be combined or performed in any othercomponent.

It should be noted that while FIG. 7 is illustrated as a distributedsystem, in alternative examples, components of the vehicle 702 can beassociated with the computing device(s) 730 and/or components of thecomputing device(s) 730 can be associated with the vehicle 702. That is,the vehicle 702 can perform one or more of the functions associated withthe computing device(s) 730, and vice versa.

EXAMPLE CLAUSES

A. A system comprising: one or more processors; and one or morenon-transitory computer readable media storing instructions executableby the one or more processors, wherein the instruction, when executed,cause the system to perform operations comprising: collecting schedulingdata associated with a process from a scheduling component associatedwith the first processing resources; generating a first time stampassociated with the scheduling data based on a local clock signals;collecting execution data associated with the process from theprocessing resource during execution of the process; generating a secondtime stamp associated with the execution data based on the local clocksignals; collecting completion data associated with the completion ofthe process by the processing resource; generating a third time stampassociated with the completion data based on the local clock signals;receiving substantially simultaneously a local reference clock signaland a global reference clock signal, the local reference clock signalassociated with the first processing resource and the global referenceclock signal associated with the system; determining a differencebetween the local reference clock signal and the global reference clocksignal; adjusting the first time stamp, the second time stamp, and thethird time stamp based at least in part on the difference; generating atleast one message including at least one of the scheduling data, theexecution data, or the completion data; and causing the one or morecommunication connections to send the message to a remote system via anetwork.

B. The system of paragraph A, the operations further comprising:determining an execution threshold for the process based at least inpart on the speed of the vehicle and the scheduling data, executiondata, and completion data associated with the process executing on thefirst processing resource.

C. The system of paragraph A, the wherein determining a time associatedwith the execution of the process by the processing resource exceeds anexecution threshold; and sending, in response to determining the timeexceeded the execution threshold, an alert to an operational system ofthe vehicle.

D. A method comprising: receiving, by a metric collection component, afirst indication of an initialization of a process on a processingresource; determining, by the metric collection component, first timedata associated with the process initialization on the processingresource; receiving, by the metric collection component, a secondindication of a process completion by the processing resource;determining, by the metric collection component, second time dataassociated with the process completion by the processing resource;receiving substantially simultaneously a local reference clock signaland a global reference clock signal, the local reference clock signalassociated with the processing resource and the global reference clocksignal associated with a system including the processing resource and atleast one additional processing resource; determining a differencebetween the local reference clock signal and the global reference clocksignal; adjusting the first time data and the second time data based atleast in part on the difference; generating a message comprising dataassociated with the process, the first time data, and the second timedata; and sending the message to a remote system.

E. The method of paragraph D, further comprising determining anexecution threshold for the process based at least in part on a speed ofa vehicle associated with the processing resource and the dataassociated with the process.

F. The method of paragraph D, wherein the data associated with theprocess comprises one or more of an identifier associated with theprocessing resource.

G. The method of paragraph D, the data associated with the processcomprises a total processing time, one or more process dependencies, oneor more cache hits associated with the process, and one or more cachemisses associated with the process.

H. The method of paragraph D, further comprising: determining a timeassociated with an execution of the process by the processing resourceexceeds an execution threshold, the execution threshold variable basedon a speed of a vehicle including the processing resource; and sending,in response to determining the time exceeded the execution threshold, analert to an operational system of the vehicle.

I. The method of paragraph H, further comprising reducing, by theoperational system, a speed of the vehicle in response to the timeexceeding the execution threshold.

J. The method of paragraph D, further comprising: receiving, by themetric collection component, a third indication of an initialization ofa second process on a second processing resource, the second processingresource different than the first processing resource; determining, bythe metric collection component, third time data associated with thesecond process initialization on the second processing resource;receiving, by the metric collection component, a fourth indication of aprocess completion of the second processing resource by the secondprocessing resource; determining, by the metric collection component,fourth time data associated with the second process completion by thesecond processing resource; receiving substantially simultaneously asecond local reference clock signal and the global reference clocksignal, the second local reference clock signal associated with thesecond processing resource; determining a second difference between thesecond local reference clock signal and the global reference clocksignal; adjusting the third time data and the fourth time data based atleast in part on the second difference; generating a second messagecomprising data associated with the second process, the third time data,and the fourth time data; and sending the second message to the remotesystem.

K. The method of paragraph D, further comprising: receiving, by themetric collection component, a third indication associated with theexecution of the process from the processing resource; determining, bythe metric collection component, third time data associated with theexecution of the process; and wherein the message comprises the thirdtime data.

L. The method of paragraph D, wherein the metric collection component ispart of a kernel associated with the processing resource.

M. The method of paragraph D, further comprising sending the alert to aremote operator associated with the autonomous vehicle.

N. The method of paragraph D, wherein the process is associated with oneor more of localization, prediction, or perception of the autonomousvehicle.

O. A non-transitory computer-readable medium storing instructions that,when executed, cause one or more processors to perform operationscomprising: identifying an individual process executing on theprocessor, the individual process associated with normal operation of anautonomous vehicle; capturing data associated with the execution of theindividual process; receiving schedule data associated with the one ormore processors; determining that the execution of the individualprocess has exceeded an execution threshold; generating an alert basedat least in part on the data associated with the execution of theindividual process and the schedule data; and sending the alert to asystem configured to make operational decisions for the autonomousvehicle.

P. The non-transitory computer-readable medium of paragraph O, whereinthe execution threshold is variable based on a speed of the autonomousvehicle.

Q. The non-transitory computer-readable medium of paragraph O, whereinthe process is associated with one or more of localization, prediction,or perception.

R. The non-transitory computer-readable medium of paragraph O, whereinthe execution data include first time data and the operations furthercomprising: receiving substantially simultaneously a local referenceclock signal associated with the processor and a global reference clocksignal associated with the autonomous vehicle; determining a differencebetween the local reference clock signal and the global reference clocksignal; and adjusting the time data based at least in part on the seconddifference.

S. The non-transitory computer-readable medium of paragraph O, theoperations further comprising: reducing a speed of the vehicle inresponse to the execution threshold being exceeded.

T. The non-transitory computer-readable medium of paragraph O, furthercomprising sending the alert to a remote system via one or morenetworks.

While the example clauses described above are described with respect toone particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses can also beimplemented via a method, device, system, a computer-readable medium,and/or another implementation. Additionally, any of examples A-T may beimplemented alone or in combination with any other one or more of theexamples A-T.

CONCLUSION

As can be understood, the components discussed herein are described asdivided for illustrative purposes. However, the operations performed bythe various components can be combined or performed in any othercomponent. It should also be understood, that components or stepsdiscussed with respect to one example or implementation may be used inconjunction with components or steps of other examples. For example, thecomponents and instructions of FIGS. 5 and 6 may utilize the processesand flows of FIGS. 1-3 .

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein can be presentedin a certain order, in some cases the ordering can be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations described herein need not beperformed in the order disclosed, and other examples using alternativeorderings of the computations could be readily implemented. In additionto being reordered, in some instances, the computations could also bedecomposed into sub-computations with the same results.

What is claimed is:
 1. A vehicle comprising: a first processingresource; one or more communication connections; one or morenon-transitory computer readable media storing instructions, that whenexecuted by one or more processors, cause the one or more processors toperform operations comprising: collecting scheduling data associatedwith a process from a scheduling component associated with the firstprocessing resource, wherein the process is associated with one or moreof localization, prediction, or perception; generating a first timestamp associated with the scheduling data based on a local clocksignals; collecting execution data associated with the process from thefirst processing resource during execution of the process; generating asecond time stamp associated with the execution data based on the localclock signals; collecting completion data associated with completion ofthe process by the first processing resource; generating a third timestamp associated with the completion data based on the local clocksignals; receiving substantially simultaneously a local reference clocksignal and a global reference clock signal, the local reference clocksignal associated with the first processing resource and the globalreference clock signal associated with the vehicle; determining adifference between the local reference clock signal and the globalreference clock signal; adjusting the first time stamp, the second timestamp, and the third time stamp based at least in part on thedifference; generating at least one message including at least one ofthe scheduling data, the execution data, or the completion data; andcausing the one or more communication connections to send the message toa remote system via a network.
 2. The vehicle of claim 1, wherein theoperations further comprise: determining an execution threshold for theprocess based at least in part on speed of the vehicle and thescheduling data, execution data, and completion data associated with theprocess executing on the first processing resource.
 3. The vehicle ofclaim 1, wherein the operations further comprise: determining a timeassociated with the execution of the process by the first processingresource exceeds an execution threshold; and sending, in response todetermining the time exceeded the execution threshold, an alert to anoperational system of the vehicle.
 4. A method comprising: receiving, bya metric collection component, a first indication of an initializationof a process on a processing resource, wherein the process is associatedwith one or more of localization, prediction, or perception;determining, by the metric collection component, first time dataassociated with the initialization on the processing resource;receiving, by the metric collection component, a second indication of aprocess completion by the processing resource; determining, by themetric collection component, second time data associated with theprocess completion by the processing resource; receiving substantiallysimultaneously a local reference clock signal and a global referenceclock signal, the local reference clock signal associated with theprocessing resource and the global reference clock signal associatedwith a system including the processing resource and at least oneadditional processing resource; determining a difference between thelocal reference clock signal and the global reference clock signal;adjusting the first time data and the second time data based at least inpart on the difference; generating a message comprising data associatedwith the process, the first time data, and the second time data; andsending the message to a remote system.
 5. The method of claim 4,further comprising determining an execution threshold for the processbased at least in part on a speed of a vehicle associated with theprocessing resource and the data associated with the process.
 6. Themethod of claim 4, wherein the data associated with the processcomprises one or more of an identifier associated with the processingresource.
 7. The method of claim 4, wherein the data associated with theprocess comprises a total processing time, one or more processdependencies, one or more cache hits associated with the process, andone or more cache misses associated with the process.
 8. The method ofclaim 4, further comprising: determining a time associated with anexecution of the process by the processing resource exceeds an executionthreshold, wherein the execution threshold is based on a speed of avehicle including the processing resource; and sending, in response todetermining the time exceeded the execution threshold, an alert to anoperational system of the vehicle.
 9. The method of claim 8, furthercomprising reducing, by the operational system, a speed of the vehiclein response to the time exceeding the execution threshold.
 10. Themethod of claim 4, wherein the processing resource is a first processingresource, the method further comprising: receiving, by the metriccollection component, a third indication of an initialization of asecond process on a second processing resource, the second processingresource different than the first processing resource; determining, bythe metric collection component, third time data associated with theinitialization of the second process on the second processing resource;receiving, by the metric collection component, a fourth indication of aprocess completion of the second processing resource by the secondprocessing resource; determining, by the metric collection component,fourth time data associated with the process completion of the secondprocess by the second processing resource; receiving substantiallysimultaneously a second local reference clock signal and the globalreference clock signal, the second local reference clock signalassociated with the second processing resource; determining a seconddifference between the second local reference clock signal and theglobal reference clock signal; adjusting the third time data and thefourth time data based at least in part on the second difference;generating a second message comprising data associated with the secondprocess, the third time data, and the fourth time data; and sending thesecond message to the remote system.
 11. The method of claim 4, furthercomprising: receiving, by the metric collection component, a thirdindication associated with execution of the process from the processingresource; determining, by the metric collection component, third timedata associated with the execution of the process; and wherein themessage comprises the third time data.
 12. The method of claim 4,wherein the metric collection component is part of a kernel associatedwith the processing resource.
 13. The method of claim 4, furthercomprising sending an alert to a remote operator associated with anautonomous vehicle.
 14. One or more non-transitory computer-readablemedia storing instructions that, when executed, cause one or moreprocessors to perform operations comprising: identifying an individualprocess executing on the processor, the individual process associatedwith normal operation of a vehicle, wherein the individual process isassociated with one or more of localization, prediction, or perception;capturing data associated with the execution of the individual process;receiving schedule data associated with the one or more processors;determining that the execution of the individual process has exceeded anexecution threshold; generating an alert based at least in part on thedata associated with the execution of the individual process and theschedule data; and sending the alert to a system configured to makeoperational decisions for the vehicle.
 15. The one or morenon-transitory computer-readable media of claim 14, wherein theexecution threshold is variable based on a speed of the vehicle.
 16. Theone or more non-transitory computer-readable media of claim 14, whereinthe data associated with the execution of the individual process includetime data and the operations further comprising: receiving substantiallysimultaneously a local reference clock signal associated with theprocessor and a global reference clock signal associated with thevehicle; determining a difference between the local reference clocksignal and the global reference clock signal; and adjusting the timedata based at least in part on the difference.
 17. The one or morenon-transitory computer-readable media of claim 14, the operationsfurther comprising: reducing a speed of the vehicle in response to theexecution threshold being exceeded.
 18. The one or more non-transitorycomputer-readable media of claim 14, further comprising sending thealert to a remote system via one or more networks.